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Pulse!!  The  Virtual  Clinical  Learning  Lab 
and  Center  of  Excellence 

Claudia  L.  McDonald,  Ph.D. 

Associate  Vice  President  for  Special  Projects 
Principal  investigator 

Texas  A&M  University-Corpus  Christi 
Office  of  Special  Projects 
Center  for  Virtual  Medical  Education 

a.  Scientific  and  Technical  Objectives  (limit  200  vvords/183) 

The  scientific  and  technical  objectives  of  Pulse!!  The  Virtual  Clinical  Learning  Lab  are 
consistent  with  the  original  proposal  (2005);  viz.,  to  test  whether  game-based  technologies 
provide  learning  environments  for  the  acquisition  of  clinical  knowledge  and  critical  thinking 
leading  to  differential  diagnostic  skills  for  the  practice  of  medicine. 

The  program  has  been  built  with  an  iterative,  play-based  design  that  allows  for  continual 
prototyping  and  developing  in  a  cyclical  loop.  This  provides  flexibility  that  allows  creation  of 
newr  scenarios  replicating  the  changing  world.  Selected  patient  physiological  traits  have  been 
created,  and  complex  object  functions  have  been  scripted,  as  have  collision  information,  artificial 
intelligence,  path-finding  and  the  use  of  motion-capture  data. 

A  further  technical  objective  was  creation  of  case-authoring  capabilities  for  use  bv  game 
developers  and  medical  faculty.  The  Case  Authoring  Tool  (CAT)  allows  modification  of  existing 
cases  and  creation  of  new  cases  that  utilize  catalogs  of  virtual  assets  -  patients,  patient  parts, 
equipment,  environments  and  personnel  -  and  physiological  parameters  reflecting  a  variety  of 
case-based  states.  The  Case  Authoring  Tool  also  comprises  tutoring  functions  that  replicate 
clinical  mentoring  in  the  process  of  developing  critical  thinking  and  diagnostic  skills. 

b.  Approach  (limit  200  words/189) 

The  fundamental  approach  of  the  Pulse!!  project  toward  its  scientific  and  technical  goals 
has  been  that  of  creating  a  curricular  delivery  system  for  medical  case  interfaced  with 
construction  of  virtual  environments,  physiological  assets  and  case-authoring  tools  using  state- 
of-thc  art  technologies  common  to  the  videogame  industry  but  here  appropriated  for  graduate 
medical  education  and  allied  health-care  training.  The  development  process  has  been  integrated 
with  a  rigorous  evaluation  regimen  to  ensure  the  platform's  reliability  and  validity  as  an 
environment  within  which  medical  and  hcalth-care  learning  can  occur. 

This  case-based  approach  has  incorporated  subject-matter  experts'  expertise  in  the  fields 
of  medicine,  computer-based  virtual-reality,  human  factors  and  learning  theory,  all  invested 
toward  production  of  a  virtual-world  experience  focused  on  skill  sets  to  be  mastered  by  health¬ 
care  professionals.  Case  development  has  adhered  to  accepted  protocols  of  medical  training  and 
education,  including  the  Accreditation  Council  for  Graduate  Medical  Education  (ACGME).  Pre¬ 
hospital  Trauma  Life  Support  (PTLS)  and  its  subset.  Tactical  Combat  Casualty  Care  (TCCC), 
among  others. 

Case-development  has  proceeded  from  the  exterior  body  to  interior  anatomical  processes 
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due  to  the  likelihood  that  interior  processes  (e.g.,  fluid  dynamics)  arc  beyond  the  current  reach  of 
the  videogame  industry. 

c.  Concise  Accomplishments  (limit  200  words/170) 

Concise  accomplishments  of  deliverable  demonstrations  for  this  grant  period  include  the 
following. 

•  Case-authoring  tool  (CAT):  This  component  provides  a  broad  range  of  modifications 
to  existing  cases.  Work  completed  includes  automation  of  asset  management;  design 
and  development  of  a  user-interface  editor;  design  and  development  of  a  physiology 
editor;  collection  of  user  feedback  on  the  toolkit  and  implementation  of 
improvements  based  on  user  feedback. 

•  3-D  Scene  Designer:  This  component  increases  the  ability  of  developers  to  automate 
asset  generation.  Work  completed  includes  enhancement  of  the  environment  editor; 
testing  of  the  automated  asset  tool;  collection  of  user  feedback  and  implementation  of 
suggested  improvements. 

•  Tutoring  system:  This  component  provides  automatic  feedback  to  users  as  they  work 
cases.  Work  completed  includes  refining  the  feedback  system  and  enhancing  the 
capabilities  and  accuracy  of  real-time  task  measurement  and  evaluation. 

•  Case-authoring  assets:  This  task  greatly  increases  options  available  to  medical 
instructors  in  building  their  own  cases.  Work  completed  includes  creation  of  new 
environments,  including  medical  staff,  patients,  medical  equipment  and  procedures; 
and  research  and  design  of  new  physiology  assets. 

d.  Expanded  Accomplishments:  The  Pulse!!  Evaluation  Plan  (no  word  limit) 

Verification  and  validation  of  the  platform  has  been  an  integral  part  of  the  design  and 
development  of  Pulse!!  Human  performance,  instructional,  human  factors  and  evaluation  experts 
are  involved  in  all  phases  of  Pulse! !  design  and  helped  drive  crucial  design  decisions.  This 
approach  has  been  truly  iterative  as  prototype  versions  of  the  system  were  made  available, 
evaluated  and  resulting  data  fed  back  to  improve  design.  Moreover,  a  multi-phase  summative 
and  formative  evaluation  plan  was  devised  and  has  been  implemented  since  the  program's 
inception.  The  plan  included  multipurpose  evaluation  activities,  and  it  was  designed  to 
determine  the  reliability  and  validity  of  the  Pulse!!  platform  as  a  medium  for  medical  education 
and  training.  The  multi-component  evaluation  strategy  was  shaped  to  measure  outcomes.  Results 
indicate  that  the  Pulse!!  learning  platform  is  reliable  and  valid  as  a  delivery  system  and  teaching 
strategy  for  curricula  in  the  health  professions. 

There  have  been  10  studies  at  five  sites,  including:  National  Naval  Medical  Center 
Bethesda,  Yale  University  School  of  Medicine,  The  Johns  Hopkins  School  of  Medicine.  Baystate 
Medical  Center-Tufts  University  School  of  Medicine  and  the  Emergency  Medicine  Residency 
Program  at  CHRISTUS  Spohn  Memorial  Hospital,  an  affiliate  of  the  Texas  A&M  University 
Health  Science  Center.  Five  physicians,  one  at  each  site,  were  co-primary'  investigators  for 
evaluation  of  the  Pulse!!  platform.  The  sample  (n=100)  was  composed  of  senior  physicians, 
fellows,  residents  and  medical  students.  Power  analysis  for  a  2x2  chi-squared  statistic  was 
calculated  using  a  power  level  of  .80  and  an  estimated  effect  size  of  .50.  The  required  sample 
size  indicated  by  the  power  analysis  was  either  met  or  exceeded  for  each  evaluation  category'. 
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Results  from  all  studies  found  strong,  significant  effects  that  support  the  use  of  the  Pulse!! 
learning  platform;  thus,  the  studies’  results  can  he  viewed  with  confidence  that  the  platform  is  a 
reliable  and  valid  teaching/training  tool. 

The  evaluation  plan  called  for  several  categories  to  be  assessed  with  methods  other  than 
research  studies.  Components  assessed  were  system  design,  learning  features,  usability  and 
training  effectiveness,  as  noted  below  with  analytical  method  and  outcomes. 

System  desiun 

Assessing  the  soundness  of  system  design  included  a  variety  of  evaluation  questions  to 
ensure  that  the  system  was  developed  as  planned  and  can  be  easily  used  by  trainees.  Aspects 
included  determining  whether  the  underlying  physiological  models  were  valid  and  whether  they 
interact  with  one  another  in  a  realistic  way;  comparing  system  functionality  to  design 
specifications  to  ensure  that  the  platform  performs  as  intended;  and  ensuring  that  basic  usability 
issues  are  resolved. 

Based  on  a  thorough  analysis  of  the  system  design  approach  and  resulting  system,  it  was 
verified  that  the  Pulse  !!  platform  was  developed  in  accordance  with  system  specifications. 
Physiological  and  mathematical  models  were  subjected  to  an  iterative  review  process  that 
included  several  subject  matter  experts  until  it  was  determined  that  the  models  were  accurate  and 
sufficiently  robust  to  meet  the  project's  goals. 

Learning  features 

Assessing  the  degree  to  which  the  system  incorporated  sound  learning  features  uses 
extant  literature  in  the  science  of  learning,  synthetic  experience,  game-based  learning  and 
scenario-based  training  as  a  guide  to  determine  that  the  system  embodies  sound  learning 
strategies  and  features.  These  included  the  following: 

•  Specific,  measurable  learning  objectives; 

•  Appropriate  scenario  design,  including  trigger  events; 

•  Sound  instructional  strategies/elements; 

•  Accurate  performance  assessment  and  data  recording  strategies; 

•  Sound  diagnosis  routines  and  models; 

•  Appropriate  feedback  and  after-action  review  strategies. 

Early  in  the  development  process,  several  mechanisms  were  put  in  place  to  ensure  that 
learning  objectives  were  appropriately  addressed  in  design.  This  resulted  in  matrices  that  lay  out 
learning  objectives  and  then  specify  in  detail  with  task-analyses  how  objectives  are  represented 
in  the  platform.  Working  with  project  personnel,  subject  matter  experts  and  programmers,  the 
evaluation  team  helped  guide  scenario  design  to  ensure  that  appropriate  cues  were  available  to 
learners  as  they  worked  through  the  cases. 

Usability 

Establishing  the  usability  of  the  system  includes  both  the  effectiveness  of  the  user 
interface,  i.e..  how  easy  it  is  to  learn,  how  easy  it  is  to  navigate,  how  long  it  takes  to  master  and 
the  quality  of  tutorial  material;  and  "playability.”  i.e..  the  extent  to  which  the  training  provides  a 
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pleasurable,  challenging,  fun  environment  for  learning.  This  aspect  is  crucial  to  the  idea  that 
learners  will  be  intrinsically  motivated  to  engage  in  the  instruction  because  it  has  inherently 
motivating  qualities. 

Usability  testing  proceeded  down  two  paths.  First,  the  basic  question  of  whether  users 
would  be  able  to  use  the  system  easily,  find  it  intuitive.  To  address  this  question,  a  usability 
study  with  subjects  from  two  teaching  hospitals  in  the  Northeast  was  conducted.  Subjects  were 
drawn  from  a  variety  of  medical  specialties  and  experience  levels,  from  medical  students  through 
senior  physicians,  to  ensure  that  the  interface  was  usable  across  a  variety  of  learner  and 
experience  levels.  Through  systematic  observation,  trained  evaluators  concluded  that  the  basic 
interface  was  usable  and  intuitive. 

A  “playability”  study  from  a  third  teaching  hospital  was  designed  to  test  specific  aspects 
of  the  interface.  In  this  case,  the  evaluators  gave  each  participant  a  specific  set  of  tasks  to 
complete  within  the  platform.  The  participants'  ability  to  complete  each  task  was  timed  and 
outcomes  noted.  These  data,  along  with  information  from  earlier  studies  were  fed  back  to 
developers  so  that  appropriate  improvements  to  the  interface  could  be  completed.  The  results 
support  the  Pulse!!  platform  as  a  viable  environment  in  which  to  embed  instruction.  Participants 
responded  well  to  the  technology  and  were  able  to  use  the  system  quickly. 

Training  effectiveness 

A  modified  version  of  Kirkpatrick's  hierarchy,  used  extensively  throughout  industry  and 
the  military  to  assess  training  effectiveness,  provides  the  framework  to  assess  the  following 
elements. 

Pre-training  assessment  determines  the  degree  to  which  the  trainee  is  prepared  to  benefit 
from  the  training.  Research  has  shown  that  trainees  who  lack  the  cognitive  or  affective 
prerequisites  for  training  do  not  perform  as  well  as  trainees  who  are  better  prepared.  Assessing 
readiness  for  training  can  indicate  whether  trainees  need  remedial  attention  prior  to  entering 
training  or  how  training  might  be  tailored  to  specific  needs;  e.g.,  by  starting  with  less  challenging 
scenarios.  The  sample  (n=  1 00)  was  screened  for  participation  through  the  collection  of 
demographic  data  by  survey  instruments  administered  prior  to  training.  These  data  were 
evaluated  by  subject-matter  experts. 

Reactions/motivation  measures  are  the  most  common  method  of  measuring  training 
effectiveness  but  not  necessarily  the  most  informative.  Research  has  shown  that  simple  reaction 
measures  are  not  necessarily  correlated  with  actual  learning.  This  does  not  mean  that  motivation 
is  unimportant;  only  that  reaction  measures,  by  themselves,  are  insufficient.  In  addition, 
reactions  that  involve  trainees'  beliefs  about  the  value  or  utility  of  training  have  been  shown  to 
predict  performance  better  than  affective  reactions.  Hence,  measuring  reactions  was 
recommended  for  the  Pulse!!  project. 

Reactions/motivation.  Reaction  data  on  the  same  sample  was  collected  to  determine 
whether  users  would  find  the  system  to  be  useful  and  engaging.  Results  indicated  that  the 
overwhelming  majority  (82%)  of  participants  reacted  positively  to  using  the  Pulse!!  platform. 
Over  80%  also  reported  that  engaging  in  the  platform  held  their  interest,  and  all  but  one 
respondent  reported  that  the  platform  was  visually  appealing. 

Learning,  defined  as  permanent  cognitive  change,  is  the  second  most  popular  training 
effectiveness  measure,  and  it  is  typically  assessed  via  paper  and  pencil  tests;  and  in  this  case, 
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through  one  of  several  candidate  strategies  to  assess  expert  know  ledge  organization.  Before  a 
learning  study  could  be  completed  it  was  first  necessary  to  validate  metrics  that  would  be  used  to 
measure  performance.  To  do  this,  an  “expert  referent’'  was  compiled  using  input  from  subject- 
matter  experts.  The  expert  referent  was  then  used  to  create  calibrated  videotapes  of  four  levels 
of  performance,  which  were  presented  to  thirteen  faculty  physicians  at  two  teaching  hospitals  in 
the  Northeast  (all  participants  had  extensive  experience  in  trauma  management).  Specifically, 
one  tape  showed  nearly  perfect  performance,  two  showed  performance  that  required 
improvement,  and  one  showed  performance  that  was  unacceptable.  The  question  was  whether 
faculty  physicians  would  correctly  distinguish  the  different  levels  of  performance.  Results 
indicated  a  high  level  of  agreement  in  how  participants  rated  performance  in  the  tapes.  This 
provided  evidence  that  the  expert  referent  was  valid  and  could  be  used  as  a  basis  to  assess 
performance  in  learning  studies. 

T  he  initial  test  of  learning  effectiveness  was  conducted  on  medical  students  and  residents 
from  teaching  hospitals  in  the  Northeast  and  South.  Results  indicated  that  training  trauma  life- 
support  procedures  using  the  Pulse!!  platform  (with  appropriate  feedback)  is  superior  to  paper- 
based  learning  methods,  in  which  students  read  what  an  expert  would  do  and  mentally  simulated 
the  scenarios.  For  example,  students  who  experienced  the  Pulse!!  platform  were  less  likely  to 
commit  errors  involving  the  airway  ( X 2  -  5.94,  /?  <  .05);  breathing  (X2  -  5.84,  p  <  .05); 

Disability  ( X' 2  =  10. 96,  p  <  .01);  and  secondary  survey  (X2  =  5.05.  p  <  .05).  Pulse!!  users  also 
made  fewer  total  errors  ( t  =  -4.44,/?  <  .05).  Moreover,  statistically  significant  differences  were 
found  regardless  of  the  data  collection  site  or  demographics  of  the  learner  (/  =  0.67.  p  n.s.). 
Distribution  analyses  indicated  no  significant  differences  in  the  distribution  of  errors  among  any 
of  the  sites  (F=  1.11./?  =  n.s.). 

Transfer  of  training,  learning  and  training  behavior  is  a  complex  phenomenon  to 
measure.  Data  are  collected  from  a  variety  of  sources  (e.g.,  supervisors,  peers,  archival  records, 
etc.)  and  augmenting  quantitative  measures  with  surveys  determine  whether  barriers  to 
performance  exist  in  the  operational  setting.  Training  behavior  refers  to  the  extent  to  which  the 
trainee  can  demonstrate  mastery  of  crucial  skills.  Training  behavior  is  typically  measured 
through  work  sample  tests  or  simulations.  The  nature  of  the  Pulse!!  learning  system  means  that 
this  kind  of  assessment  can  be  done  relatively  easily  b>  preparing  several  test  scenarios  that 
contain  no  hints,  feedback  or  other  performance  enhancing  features. 

The  study  hypothesis  was  that  subjects  who  receive  trauma  management  training  in 
Pulse!!  will  outperform  those  who  receive  standard  paper-based  training.  Researchers  used  a  chi- 
square  analysis  with  subjective  impressions  to  assess  the  likelihood  of  errors  in  trauma 
management  as  a  function  of  the  training  group  (Pulse!!  v.  paper-based  training).  Participants 
across  four  sites  who  received  trauma  management  training  in  Pulse!!  outperformed  those  who 
received  paper-based  training.  Pulse!  [-trained  participants  made  significantly  fewer  errors  on 
trauma  management  procedures. 

Conclusion 

Based  on  significant  and  robust  findings.  Pulse!!  has  been  shown  to  be  a  reliable,  valid 
teaching  tool  that  can  be  used  across  the  spectrum  of  personnel  in  Navy  medicine,  education  and 
training.  The  learning  platform  can  be  used  with  confidence  in  the  Navy  and  Defense  health 
system. 
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e.  Work  Plan  for  Next  Reporting  Period  (limit  500  words) 

No  work  plan  has  been  developed.  The  project  is  closing  out. 

f.  Major  Problems/Issues  (limit  250  words) 

No  major  problems  or  issues  were  encountered  in  this  period  of  development. 

g.  Technology  Transfer  (limit  500  words) 

None  in  this  funding  period. 

h.  Foreign  Collaborations  and  Supported  Foreign  Nationals 

None  in  this  funding  period. 
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